Eect of Adaptive Guidance and Visualization Literacy on

Gaze Aentive Behaviors and Sequential Paerns on

Magazine-Style Narrative Visualizations

OSWALD BARRAL, SÉBASTIEN LALLÉ, ALIREZA IRANPOUR, and CRISTINA CONATI,

The University of British Columbia

We study the eectiveness of adaptive interventions at helping users process textual documents with em-

bedded visualizations, a form of multimodal documents known as Magazine-Style Narrative Visualizations

(MSNVs). The interventions are meant to dynamically highlight in the visualization the datapoints that are

described in the textual sentence currently being read by the user, as captured by eye-tracking. These in-

terventions were previously evaluated in two user studies that involved 98 participants reading excerpts of

real-world MSNVs during a 1-hour session. Participants’ outcomes included their subjective feedback about

the guidance, and well as their reading time and score on a set of comprehension questions. Results showed

that the interventions can increase comprehension of the MSNV excerpts for users with lower levels of a

cognitive skill known as visualization literacy. In this article, we aim to further investigate this result by

leveraging eye-tracking to analyze in depth how the participants processed the interventions depending on

their levels of visualization literacy. We rst analyzed summative gaze metrics that capture how users process

and integrate the key components of the narrative visualizations. Second, we mined the salient patterns in

the users’ scanpaths to contextualize how users sequentially process these components. Results indicate that

the interventions succeed in guiding attention to salient components of the narrative visualizations, espe-

cially by generating more transitions between key components of the visualization (i.e., datapoints, labels,

and legend), as well as between the two modalities (text and visualization). We also show that the interven-

tions help users with lower levels of visualization literacy to better map datapoints to the legend, which likely

contributed to their improved comprehension of the documents. These ndings shed light on how adaptive

interventions help users with dierent levels of visualization literacy, informing the design of personalized

narrative visualizations.

CCS Concepts: • Human-centered computing → User studies; Empirical studies in visualization; HCI design

and evaluation methods;

Additional Key Words and Phrases: Narrative visualization, adaptive visualization, guidance, visualization

literacy, eye-tracking, gaze metrics, scanpath, pattern mining

The reviewing of this article was managed by special issue associate editors Cristina Conati, Nava Tintarev, Davide Spano.

Oswald Barral and Sébastien Lallé contributed equally, alphabetical order.

This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC grants RGPIN-

2016-04611 and RTI-2019-00711).

Authors’ address: O. Barral, S. Lallé, A. Iranpour, and C. Conati, Department of Computer Science, The University of British

Columbia, Vancouver, BC, Canada; emails: {obarral, lalles, alireza.iranpour, conati}@cs.ubc.ca.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee

provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and

the full citation on the rst page. Copyrights for components of this work owned by others than the author(s) must be

honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists,

requires prior specic permission and/or a fee. Request permissions from permissions@acm.org.

2160-6455/2021/08-ART28 $15.00

https://doi.org/10.1145/3447992

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:2 O. Barral et al.

ACM Reference format:

Oswald Barral, Sébastien Lallé, Alireza Iranpour, and Cristina Conati. 2021. Eect of Adaptive Guidance and

Visualization Literacy on Gaze Attentive Behaviors and Sequential Patterns on Magazine-Style Narrative

Visualizations. ACM Trans. Interact. Intell. Syst. 11, 3-4, Article 28 (August 2021), 46 pages.

https://doi.org/10.1145/3447992

1 INTRODUCTION

Visualizations embedded into narrative text (Figure 1), also called Magazine-Style Narrative

Visualizations (MSNVs) [62], is one particular type of narrative visualizations widely used to

tell story with data in real-world documents (e.g., newspaper, textbook, blogs, science articles).

However, it is known that processing multimodal documents that, like MSNVs, combine text and

visual information can be challenging to some readers due to the additional visual search processes

involved in integrating the two modalities [2, 8, 23, 47, 63]. In particular with MSNV, mapping

the information provided in the text to the corresponding data points in the visualizations is a

well-known diculty [23, 33, 34, 62, 67, 72], and is even exacerbated in users with low levels of

several cognitive abilities (e.g., reading prociency, visualization literacy (viz literacy))[71]. To

alleviate this diculty, researchers have designed adaptive guidance to help users integrating the

textual information to the relevant data points in the visualizations.

Steinberger et al. [67] investigated how to guide user attention in MSNV by displaying, upfront,

a set of lines linking words in the text to the corresponding information in the visualization, lead-

ing to reduced visual search time. However, providing all of the links upfront does not scale to

visualizations that contain a large number of links between the text and the visualization, as it

may overly clutter the visualization. Zhi et al. [77] proposed an alternative approach that avoids

cluttering by allowing the users to willingly trigger the guidance mechanism, when they like. Parts

of the text that refer to datapoints in the visualization are underlined, and the user can click on

each of these parts to have the relevant elements of an accompanying visualization highlighted.

This approach, however, did not result in improved comprehension, possibly because users did not

use the self-triggered guidance in an eective manner.

In recent work [42], we attempted to overcome these issues by proposing a novel gaze-driven

adaptive mechanism that dynamically highlights the relevant parts of the visualization when users

read the corresponding part in the text (Figure 2 and sample online video

). We specically fo-

cused on MSNVs featuring bar charts, one of the most ubiquitous visualizations found in real-world

MSNV documents [16], and the guidance is displayed by dynamically thickening the border of the

bars in black. Eye-tracking is used to recognize in real-time what sentence the user is currently

reading so as to trigger the adaptation. Two user studies with 97 participants compared the out-

comes of participants who received the adaptive guidance while reading a set of 14 excerpts of

real-world MSNV, versus those of participants who underwent the same tasks without guidance

(i.e., a control group), while controlling for a set of user abilities that might inuence the outcomes.

We found in earlier work [42] that the adaptive guidance led to improved comprehension of the

MSNV excerpts for users with low levels of vis literacy—that is, the ability to use common data vi-

sualizations in an ecient and condent manner [7]. Furthermore, most users found the guidance

to be useful and easy to use, although half of them also reported that it can be moderately distract-

ing due to its dynamic nature. In this article, we leverage the eye-tracking data collected in earlier

work [42] to further investigate whether and how the adaptive guidance inuences processing

of the MSNV in a way that can explain the aforementioned eects on comprehension, perceived

usefulness, and distraction.

https://www.cs.ubc.ca/cs-research/lci/research-groups/human-ai-interaction/small_bar_higlight.avi.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:3

Fig. 1. Sample narrative visualization studied in this article, extracted from a real-world document.

Fig. 2. Sample narrative visualization with the adaptive bar highlighting mechanism from Lallé et al. [42]

(fully described in Section 3.2). The blue underlines have been added for clarity to indicate the sentence that

is inked to the highlighted bars but was not displayed to the participants.

This work is driven by previous work that leveraged eye-tracking to evaluate how users process

visualizations, by examining their underlying attentional behaviors (e.g., [10, 56, 73]). Furthermore,

previous work has showed that eye-tracking can reveal how vis literacy inuences gaze behaviors

during processing of non-adaptive visualizations, showing, for instance, that low vis literacy users

focus less on important regions of the visualizations compared to high literacy users [40]. However,

previous work did not investigate in depth how users’ attentional gaze behaviors are modied by

eective adaptive visualizations, as a way to explain how and why such adaptation improves user

performance in visualization tasks.

In this article, we start lling this gap by leveraging eye-tracking to evaluate how our gaze-

driven adaptive guidance described earlier [42] support processing of narrative visualizations,

depending on the user’s levels of vis literacy. Specically, we address the following research

questions:

• RQ1: How does adaptive guidance in the MSNV snippets inuence users’ gaze behaviors

compared to no guidance?

• RQ2: What are the changes in gaze behaviors of users with low vis literacy that can explain

their better comprehension when provided with adaptive guidance?

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:4 O. Barral et al.

We address these research questions with several complementary gaze-based analyses. The rst

analysis was presented in Barral et al. [3], in which we analyzed summative gaze metrics over

specic areas of interest (AOIs) to unveil changes in the overall user processing of the dierent

components of the visualization. In this work, we further extend this analysis by investigating

whether and how promptly users process newly triggered interventions, as a way to understand

how eective the interventions are at guiding the user’s attention to relevant data points in the

visualizations. We also look at whether MNSV order aects attention to the adaptation, to check

for a possible eect of habituation.

The second analysis, which is new in this article, complements the one from Barral et al. [3]

by investigating the users’ scanpaths (i.e., entire sequences of eye movements) to better under-

stand how the adaptive interventions inuence gaze processing. Unlike the summative metrics of

xation data used in earlier work [3], the scanpaths retain the sequential nature of eye-tracking

patterns, providing additional insights about how the users integrate dierent components of the

narrative visualizations [24, 65]. We do so by applying dierential sequence mining to the scan-

paths and show that this approach was able to uncover many additional changes in gaze behaviors

when receiving the adaptive interventions.

Finally, we look at whether there are changes in gaze behaviors of users who reported that the

adaptive guidance can be distracting, as compared to those of users who did not report distraction,

given that distraction is considered one of the main pitfalls of adaptive interventions (e.g., [19, 29,

30]).

We present four main contributions. First, we show that the interventions are successful at guid-

ing the user’s attention, as we found that a majority of the users process most of the interventions

after they appear, which is encouraging because it shows that the interventions do not go unno-

ticed and are not overly ignored by the users.

Second, we uncover the underlying eye movement changes that are induced by the target adap-

tive interventions. As discussed, although previous work provided such guidance in narrative visu-

alizations [42, 67, 77], it is still unknown how exactly users process and benet from them. We ll

this gap by showing that these adaptations lead to increased integration of the key elements of the

visualization (labels, legend, datapoints), as shown by the summative gaze metrics and the scan-

path analysis. Namely, the summative metrics reveal an increased amount of transitions between

these key elements, and the scanpath analysis indicates that the adaptive interventions gener-

ate increased back-and-forth integration between the labels and the relevant bars. Furthermore,

the summative analysis shows that the interventions lead to more processing of the relevant data-

points and of the legend of the visualization. The scanpath analysis indicates that the interventions

induce increased transitions from reading a reference sentence to processing the relevant bars. We

found no other salient behaviors among the participants who did not receive the interventions, de-

spite the fact that the reading time was the same on average regardless of whether the participant

received the interventions. This indicates that without the interventions, there was no salient at-

tentive patterns in the gaze behaviors of the participants, possibly due to more diverse processing.

Third, we reveal how this adaptive mechanism aects users dierently depending on their levels

of vis literacy. As said earlier, our previous work [42] found that users with low vis literacy were

the ones who beneted the most from the adaptive mechanism, but it was still unknown why and

how the guidance modied the way they process MSNV. We show that receiving the adaptations

leads these users to transition more often between the relevant datapoints and the legend in the

visualization, indicating improved information processing of these key elements. This constitutes

the rst analysis that investigates the relationship between users’ individual dierences and their

gaze behaviors when processing visualizations with and without dynamic adaptations.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:5

Fourth, we found that the distraction reported by about half of the participants who received the

guidance translated into a lower amount of xations on the relevant sentences in the text, possibly

because their attention was pulled away from the text by the guidance, whereas the participants

who were not distracted were able to focus more on the text. Apart from this eect, we found no

other distinctive gaze patterns among distracted and non-distracted participants, suggesting that

the perceived distraction does not deeply modify MSNV processing. This would be consistent with

previous ndings in earlier work [39] that the distraction did not lead to lower comprehension nor

lower perceived usefulness, indicating that whereas adaptive interventions can cause distraction,

this may not necessarily interfere with their eectiveness.

Based on these ndings, we discuss how to further improve the adaptive guidance, namely via

personalization to the users’ vis literacy.

2 RELATED WORK

2.1 Guidance in Multimodal Documents

Previous research has studied how to guide the user attention to relevant parts of multimodal

instructional material consisting of text and accompanying diagrams or pictures (but not visual-

izations), by means of visual cues—that is, visual prompts that guide user attention (see the work

of Van Gog [74] for an overview). These cues have been so far based on the “brushing & linking”

interaction technique [9], in which the reader selects a subsets of the data of interest in one modal-

ity (brushing), which triggers cuing of the corresponding data in the other modalities (linking). In

particular, color coding matching parts of the text and the graphics, and visual links connecting

these parts, were found to increase comprehension [31, 55]. These cues were provided either up-

front [55] or at the user’s request when clicking on a specic paragraph referring to the graphics

[31]. Brushing & linking is also frequently used in interfaces that combine multiple visualization

types—for example, to show how selecting or modifying datapoints in one visualization aects the

other ones [35, 60].

There has been recent interest in studying cuing for supporting the processing of MSNV [62].

Cuing in MSNVs is intended to guide the user attention to relevant datapoints in the visualization—

that is, datapoints that are described (or referred to) in the narrative text. One approach for deliver-

ing these cues is by displaying them up front in the MSNV, as done by Steinberger et al. [67], who

drew colored lines over the document to link words in the text to the corresponding datapoints in

the visualization. A preliminary evaluation showed that the cues can reduce task time in simple

search tasks. However, providing all cues up front is hard to scale to MSNVs with a large number of

references between the text and the visualization, as is often the case in real-world documents (e.g.,

Pew Research documents on public policy can include up to 30 references [34]), because the many

cues can visually clutter the document and overwhelm the users [1, 21, 27]. Other work allowed the

users to display the cues themselves. Specically, Zhi et al. [77] highlighted relevant datapoints in

the visualization when the users select a reference in the text, and vice versa. Although they found

that users extensively use such on-demand cues, it did not result in improved reading comprehen-

sion, possibly because users did not use the on-demand cues in an eective manner, or because not

all users can eectively process them. Metoyer et al. [48] proposed a similar approach for sports

narratives with visualizations, albeit with no evaluation. Latif et al. [43] proposed an authoring

tool to ease the implementation of visual cues in MSNVs triggered by hovering the mouse over

references, but not based on the user’s gaze.

As an alternative to cues displayed up front or on-demand, in recent work [42] we leveraged

eye-tracking to guide the user attention in an adaptive way, based on user reading behaviors

captured by an eye-tracker (see Figure 2). This guidance is also derived from the aforemen-

tioned brushing & linking interaction technique, with a main dierence compared to previous

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:6 O. Barral et al.

visualization research: the brushing part (selection of the bar to be highlighted) is performed

implicitly via gaze tracking rather than explicitly with the mouse. We found two main sets of

results in the work of Lallé et al. [42].

First, the gaze-driven adaptive guidance signicantly improved comprehension performance of

some users, depending on their levels of vis literacy. Specically, low vis literacy users achieved

signicantly higher comprehension when reading MSNVs with the adaptive guidance compared to

non-adaptive MSNVs, whereas there was no such dierence for high vis literacy users. In fact, low

vis literacy users even outperformed high vis literacy users thanks to the adaptive guidance. This

dierence in comprehension came at no expense of slower reading, as the guidance did not have

an impact on reading time. These ndings are intriguing because they indicate that gaze-driven

guidance can improve the performance of some users; however, the reasons and the underlying

processing behaviors that led to this increase in performance are still unknown. As mentioned

in Section 1, in this work we leverage eye-tracking to gain a more ne-grained understanding of

how the gaze-driven adaptation that we proposed in earlier work [42] impacts MSNV processing,

particularly for low vis literacy users who do benet from the guidance. Such analysis is also

important to understand why users with higher vis literacy do not benet from the adaptation,

by examining whether they exhibit suboptimal gaze behaviors when processing the adaptation

compared to low vis literacy users.

Second, most users reported that they found the interventions useful and easy to use (rate of 5

and above on a 7-point Likert scale); however, half of them also reported that they can be distract-

ing at times (rate of 4 and above on a 7-point Likert scale) due to their dynamicity. This distraction

remained moderate and did not escalate into hindering comprehension or perceived usefulness.

Still, it is important to study the negative impacts of this distraction and understand how to miti-

gate them. As mentioned in Section 1, as a rst step toward this direction, we examine in this work

whether and how this distraction aected how users processed the MSNVs.

2.2 Eye-Tracking for Adaptation

Eye-tracking has been used to guide the delivery of adaptation in dierent HCI applications (see

the work of Lallé et al. [40] for an overview), for instance, by triggering prompts to refocus student

attention in educational software when they look away from the screen [22], or by adapting

the content of online advertisements based on what information users look at in e-commerce

webpages [2].

In InfoVis, recent work has also used eye-tracking to drive adaptive guidance in stand-alone

(non-narrative) visualizations, although no evidence was found that these adaptations can im-

prove the user performance. Specically, Göbel et al. [22] used eye-tracking to deliver adaptive

support in maps by dynamically placing the legend of the map next to where the user is looking,

and highlighting in the legend the symbols that lie in the area of gaze location. Although they

found that users could process the adaptive legend faster than the non-adaptive one, this did not

translate into improved user performance in map reading tasks. Silva et al. [64] used eye-tracking

to recommend relevant patterns in line charts showing time-series signal, based on where the

user look at in the visualization system. A preliminary analysis revealed that users extensively

look at the recommended patterns. However, this work did not include a control group to formally

evaluate the eects of the adaptation. Unlike these works, we leverage eye-tracking to perform an

in-depth evaluation of gaze-driven adaptation, which has been shown to improve the performance

of some users, to elicit the gaze behaviors and patterns that may explain this improvement. We

further extend the previous work by providing an analysis of the users’ entire scanpaths, whereas

other works [22, 64] focused solely on how extensively the users processed the dierent parts of

the visualizations. We also extend the type of visualizations studied in other works [22, 64]by

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:7

focusing on narrative visualizations and further examine how vis literacy aects processing of the

gaze-driven guidance.

2.3 Eye-Tracking for Modeling Individual Dierences

There is extensive evidence that user performance in non-adaptive HCI and visualization tasks

is impacted by individual dierences, such as cognitive abilities, personality traits, and expertise

(see other works [37, 51, 54, 76] for overviews). To understand these dierences in performance,

researchers have leveraged eye-tracking to investigate if and how individual dierences inu-

ence processing of stand-alone visualizations. Most of this research has focused on comparing

eye-tracking behaviors of dierent groups of users, either by generating summative features of

eye-tracking or by analyzing the user’s scanpaths, as detailed next.

Summative features of eye-tracking (total number of gaze xations, mean of gaze durations, num-

ber of transitions between two areas, etc.) are meant to capture overall behaviors over the dierent

parts of the visualization. Such features have been extensively used to compare the gaze behaviors

of experts and novices in dierent visualization tasks, namely map reading [53], visual informa-

tion search [36], and processing of scientic charts [26, 68]. A few studies have investigated how

cognitive abilities and skills inuence gaze behaviors in low-level analytic tasks with bar and radar

charts [70, 73], in decision making tasks with maps and deviation charts [40], and in tasks involv-

ing understanding medical information with bar charts and line charts [52]. Results showed, in

particular, that eye-tracking can unveil suboptimal gaze behaviors exhibited by low vis literacy

users, such as performing few visual comparisons among visualizations [40] and not focusing on

important regions of the visualizations [52]. Cognitive styles were also linked to how users visu-

ally process a visualization-based bibliographic retrieval system [69] and perform visual decision-

making tasks [58]. So far, no work has looked into the relationship among adaptive guidance, gaze

behaviors, and individual dierences in visualization tasks, as we do in this work. Vis literacy has

been shown to play an important role in visualizations tasks [45]. For example, low vis literacy

was found to hinder user experience during decision-making tasks supported by maps and devi-

ation charts [40], as well as during processing of network visualizations in science museums [6]

and during visualization of medical data [52]. However, so far, few works have examined how vis

literacy inuences gaze behaviors in such tasks, with results showing that low vis literacy users

do not integrate well multiple visualizations [40], or spend less time processing the key compo-

nent of the visualization compared to high literacy users [52]. Another work examined whether

vis literacy inuences processing of visual cues displayed upfront in visualizations but found no

such inuence [33]. We extend these works by providing insights into how vis literacy inuences

the eectiveness of gaze-driven adaptive guidance in narrative visualizations.

Most of the proceeding works on eye-tracking and individual dierences have not examined

so far the context of multimodal documents, but a few works have leveraged such documents

[39, 71]. Specically, Lallé et al. [39] explored whether and how eye-tracking data relate to the

student’s motivational goals during processing of instructional material that combines text and

pictures (but not visualizations). In the work of Toker et al. [71], we studied how gaze behaviors is

aected by individual dierences in MSNVs, still with no guidance. We identied that low levels

of several cognitive abilities generate longer suboptimal processing behaviors that result in longer

reading time, such as spending longer time processing non-relevant datapoints. Here, we extend

this previous work by showing that vis literacy also inuences gaze behaviors when receiving

adaptive guidance, and we provide insights into how these behaviors explain the performance of

the users depending on their levels of vis literacy.

Users’ scanpaths have the advantage of retaining the sequential nature of the gaze over the entire

interaction. Several approaches to examine the users’ scanpaths have been studied in previous HCI

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:8 O. Barral et al.

Table 1. Summary Statistics of the References and

Relevant Bars Across the 14 Target MSNVs

MSNV Property Mean SD Min Max

Length (total words) 91 50 43 228

References 2.6 1.8 1 7

Relevant bars 10.1 7.8 2 24

studies. For instance, distance metrics have been proposed to measure the dissimilarity of scan-

paths between experts and novices in specialized tasks [11, 13, 20], and between neuro-typical and

neuro-atypical users during web browsing and information search tasks [16, 46]. Such approach,

however, requires comparing the entirety of the scanpaths, which can be computationally costly

especially in longer and open-ended tasks. As an alternative, other studies have focused on identi-

fying salient subsequences in the users’ scanpaths. For instance, the scanpaths of male and female

were compared in web search tasks by identifying the longest common subsequence in each group

[17]. Pattern-mining algorithms have been used to identify salient gaze patterns in the scanpaths

of experts versus novices in programming tasks [50] and radiology image reading tasks [44].

In InfoVis, dierential sequence mining was used to identify gaze pattern dierences among

users with high and low levels of several cognitive abilities during processing of grouped bar charts

and radar charts [66]. For instance, this study found that users with low levels of perceptual speed

exhibit signicantly less frequently patterns involving processing the labels and values of both the

bar and radar charts, as compared to their counterparts with higher levels of perceptual speed. In

this work, we use an approach similar to that of Steichen et al. [66], but to investigate how vis

literacy inuences processing of static versus adaptive visualizations. We also extend their work

[66] by applying dierential sequence mining to a user’s scanpaths when processing narrative

visualizations.

3 USER STUDIES

The eye-tracking dataset used in this work to evaluate adaptive MSNVs was previously collected

in two separate user studies that we conducted, together conforming a between-subject design. In

the rst study (fully reported in the work of Toker et al. [71]), participants read a set of MSNVs

with no adaptive interventions (control group), whereas in the second study (fully reported in the

work of Lallé et al. [42]), participants read the same MNSVs with adaptive interventions (adaptive

group). The two studies used the exact same task and procedure, as summarized next.

3.1 MSNV Set

Both the control and the adaptive study used the same set of 14 bar-chart-based MSNVs that were

derived from an existing dataset of MSNVs extracted from real-world sources (e.g., Pew Research

Center, The Guardian)[71]. Each MSNV in the dataset consists of “snippets” of larger source docu-

ments whereby each snippet includes a self-contained excerpt of the original text and one accom-

panying bar chart (see Appendix B and Figures 3 through 6). We use this format to more easily

control for dierent factors of complexity of the MSNVs that might impact their processing. In par-

ticular, the 14 MSNVs were selected to include a balanced variety of document length, and number

of referenced datapoints, as shown in Table 1. We focus on bar charts because they are one of

the most ubiquitous visualizations in real-world MSNVs [49], and we leverage dierent types of

bar chart types (four simple, six stacked, four grouped). Including visualizations types beyond bar

charts would have required many more repetitions in the experimental design, making the study

exceedingly complex and taxing for users. The selection process of the MSNV is fully described

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:9

Fig. 3. Sample MSNV with one active intervention, namely the two highlighted bars at the top of the bar

chart. The blue underlines (added for clarity but not displayed to the participants) indicates the correspond-

ing reference sentence.

in the work of Toker et al. [71]. In each MSNV, the mapping between sentences in the text that

refer to the visualization (called references from now on), and the specic referenced datapoints

in the bar chart (called relevant bars from now on), was identied via a rigorous coding process

detailed by Kong et al. [34]. Datapoints in the bar chart that are never referred to in the text are

called non-relevant bars.

3.2 Adaptive Interventions in MSNVs

For the adaptive group, we proposed in earlier work [42] the gaze-driven adaptive interventions

meant to drive the user’s attention to the relevant data in the visualization of the MSNV when

it is the most relevant (i.e., when the user is attending to that piece of information in the text).

To this end, we devised an eye-tracking mechanism that dynamically highlights the set of bars in

the MSNV visualization corresponding to the reference sentence that the user is reading. These

highlights are displayed by thickening the border of the bars in black. For instance, in Figure 3 and

Figure 4, the two bars at the top are highlighted with a thick black border when the user reads the

sentence underlined in blue in the text, as this sentence directly describes these bars. Notice that

as mentioned previously, the blue underlines in the text in Figure 3 and Figure 4 have been added

here for clarity but would not have been displayed in the interface.

As the user reads though the text, highlights corresponding to each newly read reference sen-

tence are cumulatively added to the previously highlighted bars. To help the user distinguish the

most recent highlighting from the previous ones, previous black outlines are desaturated so that

they become grey. This is shown in Figure 5 and Figure 6, where the bars thickened in black cor-

respond to the underlined reference sentence, whereas the desaturated bars at the top correspond

to the previously triggered interventions.

Importantly, we opted to not implement the opposite mechanism—that is, highlight a reference

sentence when the user starts by processing the corresponding bars in the visualization. We did

so based on preliminary analysis in our control group (see Section 3.3) that a very large majority

of our participants (>90%) reads the text rst. Thus, as a starting point, we examined the value of

guiding attention to the chart as the users read through the text, and future work can build on our

work to experiment with other forms of interventions.

This gaze-driven adaptive mechanism has been carefully designed, pilot tested, and evaluated,

as fully reported in Lallé et al. [41, 42], with most participants in the pilot and main user study

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:10 O. Barral et al.

Fig. 4. Sample MSNV with one active intervention, namely the two highlighted bars at the top of the bar

chart.

Fig. 5. Sample MSNV with one active intervention (boom two highlighted bars) and a desaturated one (top

two bars).

reporting that they found the interventions to be useful and intuitive. The experiment platform

used to trigger the adaptive interventions is publicly available on GitHub for reproducibility.

demo video of the delivery mechanism is also available online.

3.3 Procedure for Data Collection

The control study included 56 subjects (32 female), age ranging from 19 to 69 (M = 28, SD =

11), whereas the adaptive study included 63 participants (34 female), age ranging from 18 to 59

(M = 25, SD = 8). In both studies, about 60% of participants were university students. The study

procedure involves a single session lasting about 60 minutes. The session starts with the participant

lling out a consent form and undergoing calibration with the eye-tracker, a Tobii T-120 camera-

based remote eye-tracker embedded in a display of 1280 × 1024 pixels, with a sampling rate of

https://github.com/ATUAV/ATUAV_Experimenter_Platform/.

https://www.cs.ubc.ca/cs-research/lci/research-groups/human-ai-interaction/small_bar_higlight.avi.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:11

Fig. 6. Sample MSNV with one active intervention (boom three bars thickened in black) and desaturated

ones (top four bars).

120 Hz. Participants were instructed to read each of the 14 MSNVs on the computer screen. The

MSNV order was fully randomized, and there was no time limit to read the MSNVs, nor any sort

of training, to mimic how users read MSNVs in their daily life. After reading each MSNV, they

were presented with a set of comprehension questions meant to evaluate their understanding of

the MSNV they just read. These questions were based on the work by Dyson and Haselgrove

[78], where they provide ve dierent types of multiple choice questions for evaluating users’

comprehension of National Geographic articles. We designed our questionnaire based on two of

their question types that we could apply to all MSNVs in our dataset. Namely, the questionnaire

include the following:

• One title question that asks to select a suitable alternative title for the MSNV (see question

5, bottom of Figure 7) and provides a simple way to ensure that the user had a grasp of the

general document narrative.

• One or two (depending on the length of the snippet) recognition questions asking to re-

call specic information from the MSNV—that is, recognizing the name, magnitude, and/or

directionality of an entity discussed in the text and named/labeled in the bar chart (e.g.,

questions 3 and 4 in Figure 7).

In both the control and adaptive studies, the users took on average about 20 minutes to read

through all 14 MSNVs and answer the comprehension questions. Next, the Bar Chart Visualization

Literacy Test was administered to collect the levels of vis literacy of the participants [7]. This test

involves the participant answering data analysis questions based on data distributions shown in

bar charts, such as identifying the min, max, average, and trend of the distribution.

The participant

scores on the vis literacy test were retrospectively compared among the control and the adaptive

group with a Mann–Whitney U test, with results showing no signicant dierences among the two

groups (U = 1458.5, p = 0.87, r = 0.01). Last, the participants lled out a questionnaire about their

experience with the adaptive interventions, and an interview was conducted to further elicit their

response to the questionnaire. This questionnaire particularly asked participants to rate on the

Link to the test: http://peopleviz.gforge.inria.fr/trunk/vLiteracy/home/tests/bc/.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:12 O. Barral et al.

Fig. 7. Sample comprehension questionnaire used in the user studies.

Table 2. Summary Statistics of Collected Measures

Type of Measure Measure Control Group Adaptive Group

Mean SD Mean SD

Performance

Accuracy on comprehension questions 71.9% 3% 74.4% 4%

Time-on-task 56.3 sec 32 sec 60.1 sec 33 sec

Perception of the

adaptive guidance

Useful (scale: 1-7) — — 4.51 1.54

Easy to use (scale: 1-7) — — 4.8 1.42

Confusing (scale: 1-7) — — 3.1 1.66

Distracting (scale: 1-7) — — 4.3 1.83

7-point Likert scale whether they found the adaptive guidance to be useful, intuitive, confusing,

and distracting. The study procedure is fully reported in the work of Toker et al. [71]. We report

in Table 2 the summary statistics of the measures collected during the studies, across the control

and adaptive groups.

3.4 Data ality

To ensure accurate analysis, we investigated the quality of the eye-tracking data, as the triggering

mechanism requires to accurately capture series of xations over the reference sentences in the

text to display the interventions. This data quality check is extensively detailed in earlier work

[42] and is summarized next. For the purpose of this work:

—We examined the proportion of invalid gaze samples as reported by the eye-tracker, and

discarded nine users who exhibited more than 25% of missing gaze samples, as missing

xations may result in delayed or un-triggered interventions even if the participants read

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:13

the corresponding reference. This 25% threshold is commonly used in eye-tracking analysis

to ensure a high-quality dataset.

—We examined the attention map of the participant over the MSNV layout in the rst, middle,

and last task, and found that the attention map was substantially misaligned with the MSNV

layout for eight additional participants. We discarded these participants, as this technical

issue means that the participant’s xations will not be aligned with the reference sentences.

As a result, we retained 46 participants (73%) with high-quality data in the adaptive group. These

participants triggered on average 93% of the interventions (SD = 7%), thus demonstrating that the

triggering mechanism was adequate to deliver the interventions to almost all participants. The

same data cleaning process was applied to the control group to ensure a fair comparison among

the two groups, which resulted in 4 participants being discarded from the control group. This

produced a dataset of 98 participants with high data quality (52 control, 46 adaptive).

4 OVERVIEW OF ANALYSES

To address how the adaptive interventions inuence users’ gaze behaviors (RQ1), and the specic

changes in users with low vis literacy that can explain their improved performance (RQ2), we

present the following analyses:

1. We investigate whether and how promptly the participants in the adaptive group processed

the newly delivered adaptive interventions, to ascertain whether the interventions were

successful in guiding the user attention (intervention processing analysis, Section 5).

2. We leverage summative eye-tracking metrics to understand whether and how the inter-

ventions inuenced the MSNV processing, as compared to receiving no interventions (gaze

metrics comparative analysis, Section 6).

3. We investigate the users’ scanpaths by mining the salient xation patterns in the adap-

tive and control groups, to understand whether and how the interventions inuence how

users sequentially process the dierent parts of the MSNV (scanpath comparative analysis,

Section 7).

4. We leverage summative eye-tracking metrics as well as scanpaths to unveil whether there

are specic eye movements that are characteristic of users that reported being distracted by

the interventions, to identify how we could better support these users (distraction analysis,

Section 8)

In each analysis, we account for the role of vis literacy. As stated in Section 1, the second analysis

was previously reported in our previous conference paper [3], whereas the rst and the third

analyses are new in this work.

5 PROCESSING OF THE ADAPTIVE INTERVENTIONS

We examine whether, how quickly, and for how much time the participants in the adaptive group

processed the interventions, depending on their levels of vis literacy. We do so by examining the

participants’ xations after each intervention trigger and compute the following metrics for each

intervention:

• The proportion of triggered interventions for which the participant xated at least once on

the corresponding highlighted bars, as captured by the eye-tracker (proportion of processed

intervention).

• The elapsed time, in seconds, between the intervention trigger and the rst xation on the

corresponding highlighted bars (time to look at intervention).

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:14 O. Barral et al.

Table 3. Summary Statistics of Usage of the Interventions

Metric Intervention State Mean SD

Proportion of processed

intervention

Active only 49% 16%

Desaturated only 35% 15%

All 84% 18%

Time to look at intervention

Active only 10.87 sec 6.34 sec

Desaturated only 29.47 sec 11.23 sec

All 18.40 sec 9.05 sec

We also look at whether attention to the interventions is aected by MNSV order, to ascertain

whether there is an eect of habituation on intervention processing.

As described in Section 3.2, previous interventions are desaturated in grey when a new inter-

vention is triggered, so as only the most recently triggered intervention is highlighted with the

thick black border at a time. Thus, we compute the preceding metrics for two distinct sets of inter-

ventions, depending on the state of the interventions when the participant processed them for the

rst time. Namely, interventions that were processed when highlighted in black are grouped into

the active state set. Interventions that were never processed while active, but only once they were

desaturated, are grouped into the desaturated state set. Distinguishing between the two states is

meant to provide complementary insights into how users processed the interventions. Namely, the

active state captures whether the interventions caught the participants’ attention once appeared,

whereas the desaturated state reveals whether the participants ended up processing the interven-

tions only later on, after they were desaturated. Table 3 reports the summary statistics for the

three metrics in each state, as well as for both states combined (active and desaturated), over the

46 participants in the adaptive condition.

Intervention processing.Table3 (top) shows that users looked at most of the triggered interven-

tions (average 84%), and furthermore they processed on average about half of the interventions

while they were active. This is encouraging, because the primary purpose of the intervention was

to draw attention to the relevant bars in a timely manner (i.e., when the user reads the correspond-

ing piece of information). Additionally, on average, 35% of the interventions were processed only

after they were desaturated, and 16% of them were never processed, which suggests that some par-

ticipants either did not see some of the interventions when they appeared or chose to temporarily

ignore them, perhaps because they intended to nish reading a paragraph or even the entire text

rst. Although further analysis could focus on explaining this behavior, overall, the proportion of

processed interventions is substantial enough to proceed with our analysis of the impact of the

intervention on the users’ gaze behaviors.

Table 3 (bottom) shows that participants took, on average, about 10 seconds to look at the active

interventions after they were triggered, and that this time varies across participants, with a stan-

dard deviation of ∼6 seconds. Naturally, this time increased for interventions that were looked at

after they had been desaturated (around 30 seconds on average). This result suggests that inter-

ventions do not take participant’s attention away from the text immediately as they are triggered

(in fact, only 11.5% of the active interventions across all participants and tasks were looked at in

under 1 second after they appeared); rather, participants tend to take a few seconds before pro-

cessing the highlighted bars, perhaps to nish reading the reference sentence, or to process other

key elements of the MSNV such as the bar labels and legend rst.

Role of task order. As we did not include a training phase in the experimental protocol, we in-

vestigate whether we can observe any noticeable learning eect in the way users process the

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:15

adaptive visualizations over time. To do so, we run a set of Linear Mixed Models (LMMs) over

the metrics in Table 3. We use mixed models because they can handle more than one random ef-

fect at once (i.e., within-subject and within-document correlation, as each user performed several

tasks, and each of the 14 MSNVs were presented to every user). Specically, we select the metrics

listed in Table 3 one at a time as the dependent variable, with task order as the xed eect, and

Participant-ID and MSNV-ID as random eects (i.e., repeated measures). We used the lmerTest pack-

age in R [38] and account for the multiple models run (six metrics) by adjusting all p-values with the

Benjamini–Hochberg procedure to control for the false discovery rate (FDR) [5]. Results show

a signicant

main eect of task order on the proportion of ALL processed intervention (χ

(1) = 9.37,

p = .002, r = .45) with earlier tasks presenting a higher proportion of processed intervention than

later tasks, representing an average of 87% of triggered interventions being processed over tasks

1 through 7, whereas this proportion decreases to 80% on average over tasks 8 through 14. Given

that we did not include a training session, this could be related to the learning eect naturally

occurring during the interaction (i.e., loss of initial surprise eect). The lack of signicant eect of

task order on time to process intervention and time to return to text indicates that this learning eect

does not signicantly impact the way in which users process these interventions which, together

with the fact that the proportion of processed interventions remains at 80% even in the later tasks,

indicates that participants keep using and processing the interventions, suggesting that they do

not get bored or annoyed by them over time.

Role of vis literacy. To investigate whether the participant’s levels of vis literacy inuence pro-

cessing of the interventions, we run a set of LMMs over the metrics in Table 3. Similar to the

preceding task order, we select the metric listed in Table 3 one at a time as the dependent variable,

with vis literacy (low, medium, high) as the xed eect, and Participant-ID and MSNV-ID as random

eects (i.e., repeated measures). For vis literacy, we divide the participants into three bins based

on a three-way-median split of their vis literacy scores: low (N = 31, 18 control), medium (N =

34, 16 control), and high (N = 33, 18 control), following the same approach as in our earlier work

[42]. We do so because the previous work [42] found that the adaptive interventions substantially

helped users in the low bin, so much so that they signicantly outperformed users in the high bin

in the adaptive group, as well as users in the control group. We reuse the same bins throughout

this entire article, so as to elicit this previous nding, which is the goal of our research question

RQ2 as stated earlier.

Results indicate no signicant main eect of vis literacy in any of the mixed models. This in-

dicates overall that participants processed a similar amount of interventions, and with a similar

timing, regardless of their levels of vis literacy. This nding already shows that the low vis liter-

acy users did not benet from the interventions solely because they processed more of them, and

the next analyses will be focused on examining in a more ne-grained fashion how dierently

the adaptation inuenced the participants’ processing behaviors depending on their levels of vis

literacy.

6 COMPARATIVE ANALYSIS ON USERS’ GAZE METRICS OVER SPECIFIC AOIS

In this section, we describe the eye-tracking analysis on gaze metrics that we performed to address

our research questions. We rst describe the AOIs and gaze metrics used to capture users’ gaze

behaviors followed by the statistical analysis. We then discuss results pertaining to RQ1, namely

the main eect of groups found in the statistical analysis that capture changes in gaze behaviors

We report in this article statistical signicance at p < .05 after FDR correction, as well as eect sizes as high for r > .5,

medium for r > .3, and small otherwise.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:16 O. Barral et al.

Fig. 8. AOIs defined in the MSNVs.

among the adaptive and control group. Next, we examine RQ2 by discussing the eects of group

that are qualied by an interaction with vis literacy and discuss their potential implications in

terms of improved comprehension for users with low vis literacy.

6.1 AOIs and Gaze Metrics

To analyze the dierences in gaze behaviors with and without the adaptive interventions, we

dene a set of AOIs over the key regions of the MSNVs. We focus on these AOIs because in earlier

work [71], we showed that they can reveal users’ suboptimal gaze patterns within non-adaptive

MSNVs. Here we aim to understand whether the adaptive interventions modify how users attend

to them. AOIs are shown in Figure 8 and described next:

• Reference sentences (Ref) contain the combined area of all sentences that directly refer to

the visualization of the MSNV. For instance, in Figure 8, there is one reference sentence,

constituting the AOI with plain yellow border. We expect to capture more transitions from

this AOI to the visualization side, given that the adaptive interventions were designed to

guide the attention of users toward the visualization when they read a reference sentence.

• Relevant bars (R-Bar) and Non-Relevant bars (NR-Bar) contain the combined area of all bars

in the visualization that are described (R-Bar) or not (NR-Bar) by any of the references. We

expect users to focus more on the R-Bar when provided with adaptive interventions, since

the interventions specically highlight these bars, and expect the inverse for NR-Bar that

would not be highlighted.

• Legend and Labels respectively contain the legend and the combined area of all labels in the

visualization. These are key elements to integrate the information conveyed by the bars, and

we expect them to be accessed more from the R-Bar, which are highlighted by the adaptive

interventions, and less from the NR-Bar.

To evaluate how users process the MSNVs, we leverage gaze metrics meant to capture how

users allocate their attention to each of the AOIs, as well as how they integrate these AOIs by

transitioning between them.

Attention metrics. We compute the number of xations and the average xation duration within

each AOI. These two measures are complementary in understanding how users processed the

target AOI. In particular, the total number of xations gives a sense of attention allocation to

that AOI [28, 57]. The average xation duration gives a sense of how users process the AOI. In

particular, longer xations on average can indicate more cognitive processing dedicated to that

AOI [59] or that the AOI is more engaging to the users [15, 57].

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:17

Table 4. Gaze Metrics for Which a Significant Main Eect of Group Was

Found

AOI Gaze Metric Main Eect of Group

R-Bar

Number of xations

More in adaptive

(1) = 6.83, p = .0089, r = .26

Transitions to Label

More in adaptive

(1) = 6.43, p = .0011, r = .26

Label

Transitions to Legend

More in adaptive

(1) = 9.44, p = .0021, r = .31

Legend

Transitions to R-Bar

More in adaptive

(1) = 9.70, p = .0018, r = .31

Avg. xation duration

Longer in adaptive

(1) = 5.58, p = .0181, r = .24

Transition metrics. We investigate the shift in attentional focus between the dierent regions of

the visualization by computing, for each AOI, the number of transitions to all other AOIs. Namely,

a transition is dened as a series of two xations that land in two dierent AOIs. Such transitions

have been used in particular to understand the processing strategies of users—that is, how users

integrate dierent components of a visual interface [25, 57, 61]. In our case, transitions are impor-

tant to evaluate not only to what extent users process certain AOIs but also how they sequentially

access these areas to integrate the dierent parts of the visualization.

In total, we generated 6 gaze metrics for each of the ve AOIs, resulting in 30 gaze metrics.

6.2 Statistical Analysis

We evaluate dierences in the users’ attention allocation with and without adaptations using

Mixed-Eect Models, as done for the intervention processing analysis in Section 5. Specically,

we t one model for each of the 30 gaze metrics dened earlier by selecting each of them one

at a time as the dependent variable, with group (adaptive, control) as the independent variable,

vis literacy (low, medium, high) as the xed eect, and Participant-ID and MSNV-ID as random

eects (i.e., repeated measures). For average xation duration, we t an LMM using the lmerTest

package in R [38], as average xation duration follows a near normal distribution. For the num-

ber of xations and number of transitions between AOIs, we t Generalized Mixed Models

(GLMMs) for the negative binomial family using the glmer.nb function in the lme4 package in

R[4], which is suitable for discrete count distribution. We account for multiple comparisons

within each AOI family (gaze metrics within the same AOI) and report results signicant af-

ter applying the Benjamini–Hochberg FDR procedure [5]. For interaction eects of group with

vis literacy, we run post hoc, pairwise contrast comparisons using the emmeans package in R

and apply again the Benjamini–Hochberg FDR procedure to account for the multiple pairwise

comparisons.

6.3 Results

Table 4 reports the signicant main eects of group that were found on several of the gaze metrics

that we evaluated showing that the adaptive interventions inuence the users’ processing behav-

iors in several ways. Table 5 indicates the signicant interaction eect we found between group

and vis literacy, as well as the corresponding signicant pairwise comparisons.

https://CRAN.R-project.org/package=emmeans.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:18 O. Barral et al.

Table 5. Gaze Metrics for Which a Significant Interaction Eect of Group

and Vis Literacy Was Found

AOI Gaze Metric Group × Vis Literacy Signicant Pairwise Contrasts

R-Bar Transitions to Legend

(1) = 15.18,

p = .0005, r = .39

More in adaptive

(1) = 7.28, p = .0070, r = .27

Ref(×3)-RBar SS

More in adaptive

(1) = 8.43, p = .0037, r = .29

Ref(×4)-RBar SS

More in adaptive

(1) = 7.78, p = .0053, r = .28

Ref(×5)-RBar SS

More in adaptive

(1) = 9.45, p = .0021, r = .31

Ref(×6)-RBar SS

More in adaptive

(1) = 7.98, p = .0047, r = .29

Set 2

RBar-Lab-RBar APF

More in adaptive

(1) = 8.49, p = .0036, r = .29

RBar(×2)-Lab APF

More in adaptive

(1) = 7.77, p = .0053, r = .28

Lab-RBar(×2) APF

More in adaptive

(1) = 7.17, p = .0073, r = .27

Note: Successive xations are indicated in parentheses (e.g., Ref(×6) means six consecutive xa-

tions within the Ref AOI).

7.3 Results

The analysis of scanpaths yielded several main eects of group on dierent patterns for either

SS or APF metrics. The patterns for which a main eect was found are reported in Table 6.The

rst column indicates the pattern of interest, the second column indicates the metric for which a

signicant eect was found, and the third column indicates the results of the statistical tests. No

signicant interaction eect with vis literacy was found for any of the patterns tested.

We divided the patterns in Table 6 into two sets based on their similarity. The rst set, shown at

the top of Table 6, captures signicant main eects on SS for patterns involving transitions from

reference sentences (Ref) to relevant bars (RBar). The second set, shown at the bottom of Table 6,

captures signicant main eects on APF for patterns involving transitions between relevant bars

(RBar) and labels (Lab). As with the gaze metrics evaluated in Section 6, all of the main eects found

indicate that users in the adaptive group present higher values of the discriminative patterns than

users in the control group. We discuss these two sets of patterns next.

Pattern Set 1: Reference sentences (Ref) to relevant bars (RBar). This rst set of patterns involves a

series of two to six xations on reference sentences before transitioning to a relevant bar. Because

the eect for these patterns are quite similar, we solely show, in Figure 12, the eect for the rst

pattern in this set, namely “Ref(×2)-RBar.” These eects all show that these patterns are substan-

tially more frequent in the adaptive group as compared to the control group, which suggests that

the adaptive interventions are successful at directing the users’ attention toward the relevant bars

when they read a reference sentence, as captured by the repeated xations on the references. This

is encouraging since dynamically guiding the user’s attention to the appropriate bars in the charts

when it is most relevant (i.e., when the user is attending to that piece of information in the text) is

the main purpose of the adaptive interventions, as discussed in Section 3.2. These results are also

consistent with the ndings from the intervention processing analysis in Section 5,thatmostof

the interventions were processed by the participants while active.

The fact that the main eects for these patterns are only found on SS is likely due to the fact

that the MSNV snippets include few references (2.5 on average, cf. Section 3.2), and thus not so

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:23

Fig. 12. Main eect for paern set 1. Error bars show 95% confidence intervals.

Fig. 13. Main eect for paern set 2. Error bars show 95% confidence intervals.

many transitions from the references to the chart are needed within each task. The results for SS

indicate that participants in the adaptive group exhibited these patterns in more tasks (up to about

55% of the tasks for the pattern in Figure 12) compared to the participants in the control group

(33% of the tasks in Figure 12).

Interestingly, the number of transitions from Ref to R-Bar was already studied in the gaze metrics

analysis in Section 6 but did not yield any signicant main eect. Thus, the results for this set of

patterns highlight the complementarity of the gaze metrics and scanpath analysis, as we needed

to look at longer sequences to be able to capture this dierence in behavior.

Pattern Set 2: Relevant Bars (RBar) to Labels (Lab), and vice versa. The second set of patterns

involves dierent combinations of transition back and forth between relevant bars and labels,

which are found to appear, on average, more times per task in the adaptive group compared to the

control group (Figure 13). All of these main eects are found for APF but not for SS, suggesting

that the participants in both the adaptive and the control group used these patterns in a similar

amount of tasks, with the adaptive group reusing them more often within each task.

The fact that participants transitioned more frequently from the relevant bars to the labels in the

adaptive group (Figure 13(b)) is consistent with the results from the gaze metrics analysis in Section

6.3, which revealed similar ndings in terms of the number of transitions from Relevant Bars to

Labels. In addition, here we found that this transition is contextualized within longer patterns

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:24 O. Barral et al.

(either preceded or followed by another xation on the RBar AOI, see Figure 13(a) and (b)). We

also found a signicant eect for patterns comprising the opposite transition (Label to RBar, see

Figure 13(a) through (c)), which was not found in the gaze metrics analysis. These opposite patterns

are interesting because they provide additional indications that the adaptive interventions fostered

higher integration of the relevant bars and the labels, which in turn may be indicative of better

information processing on the visualizations.

As mentioned earlier, we found no interaction eect of group with vis literacy on any of the

patterns, meaning that all participants in the adaptive group exhibited higher usage of the pat-

terns found in Table 6 than the control group, regardless of their levels of vis literacy. Although

integrating the text and the visualization is a known diculty in processing MSNVs, it is still pos-

sible that these patterns are attributable to the improved performance of the low vis literacy users

in the adaptive group. Namely, the rst set of patterns may indicate that guiding the attention of

the low vis literacy users to the relevant bars help them focus on the relevant information in the

chart, while they read through the text. Similarly, the patterns in the second set could indicate that

the interventions helped the low vis literacy users going back and forth from the relevant bars to

the labels, thus integrating both. And although the interventions induced the same behaviors in

the higher vis literacy users, it is possible that high vis literacy users did not benet from them be-

cause they did not need such help, perhaps due to their already high vis literacy, although further

analysis is needed to ascertain how these patterns relate to the performance of the high and low

vis literacy users.

8 ANALYSIS ON USERS WHO WERE DISTRACTED BY THE INTERVENTIONS

In this section, we examine whether the self-reported levels of distraction caused by the inter-

ventions in the adaptive group may have inuenced the eye movements of the participants. We

especially want to ascertain whether the participants who reported some levels of distraction did

not exhibit unwanted gaze behaviors, such as ickering toward the interventions. We do so by

splitting users in two groups based on their ratings of the 7-point Likert scale item (ranging from

1 to 7) on distraction in the usability questionnaire (see Section 3.3): those who reported low levels

of distraction (ratings lower than 4, N = 21) versus those who reported moderate to high levels of

distraction (ratings greater or equal to 4, N = 25). We compare the AOI gaze metrics and scanpaths

of these two groups, depending on the levels of vis literacy of the participants.

8.1 Statistical Analysis

We borrow methodologies for Sections 5 through 7 and run three set of statistical analyses on

distraction:

1. We run a set of LMM over the metrics in Table 3, by selecting the metrics listed one at a

time as the dependent variable, with Distraction (low, high) and vis literacy (low, medium,

high) as the xed eects, and Participant-ID and MSNV-ID as random eects.

2. We run a set of Mixed Models over the summative gaze metrics described in Section 6.1,by

selecting each of the 30 metrics one at a time as the dependent variable, with Distraction

(low, high) and vis literacy (low, medium, high) as the xed eects, and Participant-ID and

MSNV-ID as random eects.

3. We mine frequent patterns in the adaptive group (SS > .4) in the same way as in Section

7.1, leading to 26 patterns (see Table A in the appendix) and run a set of Mixed Models to

evaluate AFP and SS dierences. Analogously to Section 7.1, for SS we t a GLMM for the

binomial family with a binary value indicating whether the pattern is found or not in the

scanpath of the task as the dependent variable, with Distraction (low, high) and vis literacy

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:25

Table 7. Gaze and Paern Metrics for Which Significant Eect of

Distraction Was Found

Measure Main Eect of Distraction

Average pattern frequency

Ref(×3)

Less for distracted users

(1) =12.87, p = .0003, r = .53

Average pattern frequency

Ref(×4)

Less for distracted users

(1) =12.36, p = .0004, r = .52

Number of xations

Ref

Less of distracted users

(1) = 9.31, p = .0023, r = .45

Fig. 14. Main eects found on Distraction. Error bars show 95% confidence intervals.

(low, medium, high) as the xed eects, and Participant-ID and MSNV-ID as random eects.

For APF, we t an LMM with the number of occurrences of the pattern in the scanpath of

the task as the dependent variable, Distraction (low, high) and vis literacy (low, medium,

high) as the xed eects, and Participant-ID and MSNV-ID as random eects.

8.2 Results

Table 7 presents the three main eects of distraction that were found across the three sets of

analysis that we conducted, and the eects are shown in Figure 14. No signicant interaction

eects with vis literacy were found.

We found a signicant eect of Distraction on the number of xations on the Reference sentence

AOI, indicating that distracted users do less xations on this AOI than non-distracted ones (Figure

14, left). We also found a main eect of distraction on two patterns related to consecutive xations

on these references [Ref(×3) and Ref(×4)], showing that distracted users exhibit these patterns less

than non-distracted users (Figure 14, middle and right). This indicates overall than distracted users

process slightly less extensively the reference sentences, perhaps because their attention is pulled

away from the text due to the interventions, hence their levels of reports distraction. However,

non-distracted users seem to be able to dedicate more xations to the reference sentences, possibly

because they nish reading the references instead before processing an intervention. However,

the analysis did not reveal any further signicant dierences that could help understand this be-

havior. In particular, we found no signicant dierence in how many interventions the distracted

and non-detracted users process, nor in how long they process them. This suggests that the

self-reported distraction overall has little impact on the MSNV gaze processing, which is also con-

sistent with the fact that distraction was not found to impact comprehension and reading time in

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:26 O. Barral et al.

previous work [42]. We also did not nd any interaction eect of vis literacy, which indicates that

eye movements specic to distracted users are not characterized by levels of vis literacy.

9 DISCUSSION

We rst discuss the implications that the presented eye-gaze analysis holds for evaluating adaptive

visualizations (Section 9.1). We then discuss how our results inform the design of adaptive guidance

in narrative visualizations, via personalization to the user’s levels of vis literacy (Section 9.2). Last

we discuss the main limitations of our work and how they can guide future work.

9.1 Eye-Tracking for Evaluating Adaptive Visualizations

Gaze-driven guidance to support visualization processing has recently attracted more attention

from the InfoVis research community (e.g., [22, 42, 64]). These previous works, however, have not

extensively evaluated how their adaptive guidance inuences the user’s gaze processing, as they

mostly leveraged attention maps and xation time over the adaptive component to evaluate [22,

64]. We extend these works by showing that ne-grained eye-tracking analysis can be valuable

to understand how and why the target adaptive guidance may benet specic groups of users in

narrative visualizations. Specically, we show that leveraging gaze metrics over dierent regions

of the visualization, along with the entire user scanpath, is important as well to uncover the user’s

processing strategy. Scanpaths have also been extensively used in usability studies (e.g., [14, 17,

18]); however, to the best of our knowledge, we are the rst to leverage scanpaths to evaluate the

eectiveness of gaze-driven guidance in visualization tasks.

Our results show that highlighting one specic part of the MSNV seems to generate proac-

tive processing of other important related components in the visualization, as well as increased

transitions between these components. Thus, one should leverage processing metrics (scanpaths

and transitions across AOIs) in addition to attention metrics (xation within AOIs) to evaluate in

depth the eectiveness of adaptive visualizations. Furthermore, we found that the scanpaths can

better capture how the adaptive interventions guide the users’ attention toward the highlighted

components than summative gaze metrics do, as only our scanpath analysis revealed that the in-

terventions generated increased transitions from relevant bars to the relevant bars, which was

the main purpose of the dynamic interventions in the rst place. This is especially interesting

since these transitions captured by the scanpaths are the only one that involve processing of both

modalities of the narrative visualizations (text and visualization), showing that scanpath may be

very suitable to evaluate adaptive guidance in multimodal documents.

Our results suggest that the adaptive interventions promote specic gaze strategies to process

the bar charts in the MSNVs, namely by transitioning back and forth from the relevant bars to

the labels, as well as by transitioning from the labels to the legend, and then back to the relevant

bars. These ndings are quite interesting because these are the key components of a bar chart, and

integrating these elements is crucial to fully understand the data. Interestingly, although users

spent the same amount of time reading the MSNV across the control and adaptive groups, we

found no salient gaze behaviors or scanpath patterns that the control group exhibits more than the

adaptive group. This is likely because without the interventions, participants explore the MSNVs

in more heterogeneous and unique ways and thus do not generate recurrent behaviors that could

be captured in our analysis.

We also controlled for the participants’ levels of self-reported distraction caused by the in-

terventions, which to the best of our knowledge is novel, and found that distraction generates

less xations on the reference sentences. This suggest that users distracted by the interventions

may process the references less extensively, possibly because their attention is pulled away too

soon. However, we did not nd any other results for distracted users, suggesting that overall, they

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:27

process the MSNV in a similar way. In particular, we did not nd any ickering or quick glancing

that could have indicated unwarranted behaviors. This is consistent with the fact that self-reported

distraction did not impact comprehension and reading time [39]. Nonetheless, future work could

further explore the timing of the guidance as well as other aspects that could help minimize dis-

tracting factors for these users.

9.2 Implications for Personalization

In our previous work [39], we found that when we provide adaptive guidance, vis literacy sig-

nicantly inuences comprehension, namely the guidance only improved the comprehension of

participants with low levels of vis literacy users. This suggests that there is to some extent a link

between vis literacy, comprehension, and guidance. A main goal of this work (RQ2) was to explain

this link further by examining if it can be captured in terms of salient gaze patterns that are in-

trinsic to low vis literacy users only. Our results show that providing guidance through adaptive

interventions encourages several gaze behaviors, which can be leveraged to rene the adaptation

mechanism via personalization to the user’s levels of vis literacy. We elaborate in the following

on the implications of these result, rst for users with low vis literacy and second for participants

with higher vis literacy.

Users with low vis literacy. These are the users who were found to signicantly improve their

comprehension of the MSNVs when receiving adaptive interventions in our earlier work [42], and

our analysis identied gaze behaviors that can explain their improved performance. We found that,

when provided with adaptive interventions, these users xated more on the relevant bars, transi-

tioned more between the relevant regions of the bar chart, and were more engaged in processing

the legend (Sections 6.3 and 7.3). Although all users exhibited these behaviors, it is possible that

users with low vis literacy beneted the most from them because they needed help the most due

to their low literacy, particularly their lower ability to process the salient features of the chart [7].

Furthermore, users with low vis literacy transitioned more between the legend and the relevant

bars when provided with adaptive interventions, suggesting that they better integrate these two

key elements (Section 6.3). Altogether, these changes in gaze behavior may explain the increase

in comprehension performance for these users, as mapping the relevant bars to their labels and

to the information encoded in the legend is fundamental to make sense of bar charts. This may

prove to be a challenging task for users with low vis literacy, and we provide insights into how

the adaptive interventions help them in doing so.

Although our ndings indicate that the current interventions are benecial to low vis literacy

users overall, it is still possible that some of the low vis literacy users do not perform the useful gaze

behaviors that we have identied. Moving forward, it could be worthwhile to further personalize

the guidance to these users, by encouraging them to perform these behaviors when they do not

do it spontaneously, as captured by eye-tracking. We could do so by, for instance, highlighting the

relevant labels and items in the legend, in addition to the current highlights.

Users with medium and high vis literacy. We found that the gaze behaviors of these users are

overall inuenced by the adaptive interventions, showing that they do notice and process the

highlights as much as the low vis literacy users (Sections 5, 6.3,and7.3). Importantly, these behav-

iors generated by the adaptation pertain to processing the relevant parts of the bar charts, which

was the intended goal of the adaptation, meaning that the interventions, at least, do not generate

unwarranted behaviors in these users. However, our previous work [42]showedthatproviding

these adaptations to users with medium and high vis literacy does not help them better under-

stand the visualizations. This means that the gaze behaviors we found in Sections 6.3 and 7.3 are

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:28 O. Barral et al.

not sucient to inuence the performance of these users. One possible explanation is that these

users are already able to quickly identify the relevant information in the bar chart, due to their

higher vis literacy, and they do not benet as much from improved integration of the relevant bars

with the labels and the legend. They may also not need to look at the relevant bars right away after

reading a reference sentence, possibly because they prefer to do the mapping from references to

relevant bars at their own timing. Based on these ndings, we encourage future work to investi-

gate whether these users might benet from other forms of adaptive interventions more suited to

their needs.

Altogether, our ndings show that the eectiveness of adaptive interventions in narrative vi-

sualizations could be improved via personalization to the users’ levels of vis literacy. To deliver

such personalized interventions, one possible approach is to automatically infer the users’ levels

of vis literacy from their gaze behaviors, as done in the work of Conati et al. [12]. This would

be especially suitable for narrative visualizations as our results indicate that users exhibit specic

gaze behaviors that might accurately reveal their levels of vis literacy.

9.3 Limitations

There are two main limitations in our work that can drive future research:

First, our goal of this work with RQ2 was to examine what are the salient gaze patterns that are

intrinsic to low vis literacy users only and could explain their improve comprehension as compared

to high vis literacy users. We found one such interaction eect (see Table 5), as discussed in Section

9.2, and we argue that, together with the rest of main eects of group found (see Table 4 and Table

6), they can partly explain the link between vis literacy, comprehension, and guidance. However,

this link could go beyond gaze behaviors, and there is a possibility that we are not able to fully

capture it using analysis of eye movements alone. Further research is needed to validate whether

comprehension accuracy at dierent levels of vis literacy is due to some other reasons that are not

being captured by the analysis of users’ eye movements.

Second, although eye-tracking technology is becoming more ubiquitous and accurate, there

remain limitations related to its data quality and accuracy for certain users (representing ∼27% of

users in our dataset, see Section 3.4), issues that are likely to be exacerbated outside the lab. These

limitations are potentially problematic in gaze-based adaptive systems, given that an inaccurate

gaze estimation could lead to unwanted interventions. Improving eye-tracking accuracy is outside

the scope of our research; however, in future work we should further investigate mechanisms to

better support users for which eye-tracking accuracy is low, such as the possibility to disable or

modify the behavior of the interventions, minimizing the disadvantageous eects of inaccurate

interventions.

10 CONCLUSION

In this article, we leveraged eye-tracking to evaluate the eectiveness of gaze-driven adaptive

interventions in MSNVs, a widespread form of narrative visualizations in real-world sources. The

evaluated interventions consist of dynamically highlighting the relevant datapoints (bars in a bar

chart) when users read a reference sentence that describes them, as captured by an eye-tracker.

Our previous work has shown that such adaptive interventions can improve comprehension of

the MSNV, but only for users with low vis literacy. We extend this previous work by revealing the

specic gaze behaviors that the adaptive interventions generate in users, depending on their levels

of vis literacy. In particular, providing adaptive interventions lead to overall more xations on the

relevant datapoints, longer xations on the legend, and more gaze transitions between the key

components of the visualization (datapoints, labels, legend). These results provide encouraging

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:29

evidence for this type of adaptive guidance, showing that this mechanism is eective not only in

guiding the users’ attention toward the relevant datapoints but also in facilitating processing of

the relevant information in the MSNV. In addition, we found changes specic to users with low

vis literacy, namely an increased amount of transitions from the relevant datapoints to the legend

of the visualization.

This further explains how the adaptive interventions helped users with low vis literacy bet-

ter contextualize and integrate the relevant datapoints with the rest of the components in the

visualization, leading to improved comprehension. All in all, our evaluation sheds light on the un-

derlying processing behaviors of users in adaptive narrative visualizations, driving the design of

future adaptive guidance mechanisms, more personalized to the users’ vis literacy.

APPENDIX A

Table A. Paerns Mined for the Analysis of Scanpaths

Control Adaptive

Pattern

SS APF SS APF

Lab(×3) 0.64 4.86 0.69 4.7

Lab(×2)-NRBar

0.44 0.91 0.44 0.94

Lab(×2)-RBar

0.56 1.39 0.61 1.53

Lab-NRBar(×2)

0.42 0.83 0.38 0.87

Lab-RBar(×2)

0.54 1.23 0.58 1.46

NRBar-Lab(×2)

0.46 1.01 0.47 1.05

NRBar(×2)-Lab

0.43 0.85 0.4 0.93

NRBar(×3)

0.42 2.24 0.38 1.82

Ref(×3)

0.98 74.3 1 76.93

Ref(×2) -RBar

0.33 0.52 0.53 1.09

RBar-Lab(×2)

0.56 1.39 0.66 1.72

RBar-Lab-RBar

0.36 0.64 0.44 0.86

RBar(×2)-Lab

0.55 1.2 0.61 1.53

RBar(×3)

0.6 3.33 0.61 3.76

Lab(×4)

0.49 3.15 0.51 2.96

Lab(×3)-RBar

0.43 0.83 0.43 0.8

Lab(×2)-RBar(×2)

0.39 0.68 0.42 0.77

Ref(×4)

0.97 69.42 0.99 72.07

Ref(×3)-RBar

0.3 0.45 0.5 1

RBar-Lab(×3)

0.43 0.8 0.48 0.89

RBar(×2)-Lab(×2)

0.39 0.64 0.44 0.81

RBar(×4)

0.42 1.98 0.43 2.19

Ref(×5)

0.97 65.14 0.99 67.89

Ref(×4)-RBar

0.28 0.4 0.47 0.88

Ref(×6)

0.97 61.3 0.98 64.15

Ref(×5)-RBar

0.25 0.35 0.44 0.81

Ref(×7)

0.96 57.85 0.98 60.75

Ref(×6)-RBar

0.22 0.31 0.42 0.73

Note: Patterns were of length 3–10 xations and appeared in at least 40% of

the tasks in the control or adaptive group.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:30 O. Barral et al.

APPENDIX B

This appendix includes the following:

• A screenshot of all 14 MSNVs used in the user study reported in this article.

• The set of comprehension questions participants were asked about each MSNV. Note that

the comprehension questions are asked separately right after reading the corresponding

MSNV (i.e., the participants did not see both together).

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:31

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:32 O. Barral et al.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:33

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:34 O. Barral et al.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:35

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:36 O. Barral et al.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:37

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:38 O. Barral et al.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:39

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:40 O. Barral et al.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:41

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:42 O. Barral et al.

REFERENCES

[1] Daniel Archambault, Helen C. Purchase, and Bruno Pinaud. 2010. The readability of path-preserving clusterings of

graphs. Computer Graphics Forum 29, 3 (2010), 1173–1182. DOI:https://doi.org/10.1111/j.1467-8659.2009.01683.x

[2] P. Ayres and G. Cierniak. 2012. Split-attention eect. In Encyclopedia of the Sciences of Learning,NorbetM.Seel(Ed.).

Springer, 3172–3175.

[3] Oswald Barral, Sébastien Lallé, and Cristina Conati. 2020. Understanding the eectiveness of adaptive guidance for

narrative visualization: A gaze-based analysis. In Proceedings of the 25th International Conference on Intelligent User

Interfaces (IUI ’20).ACM,NewYork,NY,1–9.DOI:https://doi.org/10.1145/3377325.3377517

[4] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2014. Fitting linear mixed-eects models using lme4.

arXiv:1406.5823.

[5] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach

to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1 (1995), 289–300.

[6] Katy Börner, Adam Maltese, Russell Nelson Balliet, and Joe Heimlich. 2016. Investigating aspects of data visualization

literacy using 20 information visualizations and 273 science museum visitors. Information Visualization 15, 3 (2016),

198–213. DOI:https://doi.org/10.1177/1473871615594652

[7] J. Boy, R. A. Rensink, E. Bertini, and J.-D. Fekete. 2014. A principled way of assessing visualization literacy. IEEE

Transactions on Visualization and Computer Graphics 20, 12 (2014), 1963–1972. DOI:https://doi.org/10.1109/TVCG.

2014.2346984

[8] Stephen Boyd Davis, Olivia Vane, and Florian Kräutli. 2016. Using data visualisation to tell stories about collections.

In Proceedings of EVA London.

[9] Andreas Buja, John Alan McDonald, John Michalak, and Werner Stuetzle. 1991. Interactive data visualization using

focusing and linking. In Proceedings of IEEE Visualization 1991. 156–163.

[10] Michael Burch, Natalia Konevtsova, Julian Heinrich, Markus Hoeferlin, and Daniel Weiskopf. 2011. Evaluation of

traditional, orthogonal, and radial tree diagrams by an eye tracking study. IEEE Transactions on Visualization and

Computer Graphics 17, 12 (2011), 2440–2448. DOI:https://doi.org/10.1109/tvcg.2011.193

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:43

[11] Nora Castner, Enkelejda Kasneci, Thomas Kübler, Katharina Scheiter, Juliane Richter, Thérése Eder, Fabian Hüttig,

and Constanze Keutel. 2018. Scanpath comparison in medical image reading skills of dental students: Distinguishing

stages of expertise development. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research and Applications.

1–9.

[12] Cristina Conati, Sébastien Lallé, Md. Abed Rahman, and Dereck Toker. 2017. Further results on predicting cognitive

abilities for adaptive visualizations. In Proceedings of the 26th International Joint Conference on Articial Intelligence.

1568–1574. DOI:https://doi.org/10.24963/ijcai.2017/217

[13] Francesco Di Nocera, Michela Terenzi, and Marco Camilli. 2006. Another look at scanpath: Distance to nearest neigh-

bour as a measure of mental workload. In Developments in Human Factors in Transportation, Design, and Evaluation,

D. de Waard, K. A. Brookhuis, and A. Toetti (Eds.). Shaker Publishing, 295–303.

[14] Andrew T. Duchowski, Jason Driver, Sheri Jolaoso, William Tan, Beverly N. Ramey, and Ami Robbins. 2010. Scan-

path comparison revisited. In Proceedings of the 2010 Symposium on Eye-Tracking Research and Applications (ETRA’10).

ACM, New York, NY, 219–226. DOI:https://doi.org/10.1145/1743666.1743719

[15] Wolfgang Einhäuser, Merrielle Spain, and Pietro Perona. 2008. Objects predict xations better than early saliency.

Journal of Vision 8, 14 (2008), 18.

[16] Sukru Eraslan, Victoria Yaneva, Yeliz Yesilada, and Simon Harper. 2019. Web users with autism: Eye tracking evidence

for dierences. Behaviour & Information Technology 38, 7 (2019), 678–700.

[17] Sukru Eraslan and Yeliz Yesilada. 2015. Patterns in eyetracking scanpaths and the aecting factors. Journal of Web

Engineering 14, 5-6 (2015), 363–385.

[18] Sukru Eraslan, Yeliz Yesilada, and Simon Harper. 2016. Eye tracking scanpath analysis techniques on web pages: A

survey, evaluation and comparison. Journal of Eye Movement Research 9, 1 (2016), 1–19.

[19] Leah Findlater and Joanna McGrenere. 2004. A comparison of static, adaptive, and adaptable menus. In Proceedings

of the SIGCHI Conference on Human Factors in Computing Systems (CHI’04). ACM, New York, NY, 89–96. DOI:https:

//doi.org/10.1145/985692.985704

[20] Mary E. Frame, Rik Warren, and Anna M. Maresca. 2019. Scanpath comparisons for complex visual search in a natural-

istic environment. Behavior Research Methods 51, 3 (2019), 1454–1470. DOI:https://doi.org/10.3758/s13428-018-1154-0

[21] Mohammad Ghoniem, J.-D. Fekete, and Philippe Castagliola. 2004. A comparison of the readability of graphs using

node-link and matrix-based representations. In Proceedings of the IEEE Symposium on Information Visualization. IEEE,

Los Alamitos, CA, 17–24. DOI:https://doi.org/10.1109/INFVIS.2004.1

[22] Fabian Göbel, Peter Kiefer, Ioannis Giannopoulos, Andrew T. Duchowski, and Martin Raubal. 2018. Improving map

reading with gaze-adaptive legends. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications

(ETRA’18). ACM, New York, NY, Article 29, 9 pages. DOI:https://doi.org/10.1145/3204493.3204544

[23] Tamara van Gog. 2014. The signaling (or cueing) principle in multimedia learning. In The Cambridge Handbook of

Multimedia Learning (2nd ed.), Richard Mayer (Ed.). Cambridge University Press, Cambridge, MA, 263–278. DOI:https:

//doi.org/10.1017/CBO9781139547369.014

[24] Joseph Goldberg and Jonathan Helfman. 2011. Eye tracking for visualization evaluation: Reading values on linear

versus radial graphs. Information Visualization 10, 3 (2011), 182–195. DOI:https://doi.org/10.1177/1473871611406623

[25] Lewis R. Goldberg. 1999. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets

of several ve-factor models. Personality Psychology in Europe 7, 1 (1999), 7–28.

[26] Joseph A. Harsh, Molly Campillo, Caylin Murray, Christina Myers, John Nguyen, and Adam V. Maltese. 2019. “Seeing”

data like an expert: An eye-tracking study using graphical data representations. Life Sciences Education 18, 3 (2019),

Article 32. DOI:https://doi.org/10.1187/cbe.18-06-0102

[27] Nathalie Henry, Anastasia Bezerianos, and Jean-Daniel Fekete. 2008. Improving the readability of clustered social

networks using node duplication. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1317–1324.

DOI:https://doi.org/10.1109/TVCG.2008.141

[28] Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost van de

Weijer. 2015. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press.

[29] Anthony Jameson and Krzysztof Z. Gajos. 2012. Systems that adapt to their users. In

The Human-Computer Interaction

Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. CRC Press, Boca Raton, FL, 431–456.

[30] Eija Kaasinen. 1999. Usability challenges in agent based services. In Intelligence in Services and Networks Paving

the Way for an Open Service Market, Lecture Notes in Computer Science, Vol. 1597, Springer, 131–142. DOI:https:

//doi.org/10.1007/3-540-48888-X_14

[31] Slava Kalyuga. 2007. Enhancing instructional eciency of interactive e-learning environments: A cognitive load

perspective. Educational Psychology Review 19, 3 (2007), 387–399. DOI:https://doi.org/10.1007/s10648-007-9051-6

[32] John S. Kinnebrew and Gautam Biswas. 2012. Identifying learning behaviors by contextualizing dierential sequence

mining with action features and performance evolution. In Proceedings of the 5th International Conference on Educa-

tional Data Mining. 57–64.

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:44 O. Barral et al.

[33] Ha-Kyung Kong, Wenjie Zhu, Zhicheng Liu, and Karrie Karahalios. 2019. Understanding visual cues in visualizations

accompanied by audio narrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

ACM, New York, NY, Article 50, 13 pages.

[34] Nicholas Kong, Marti A. Hearst, and Maneesh Agrawala. 2014. Extracting references between text and charts via

crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Los Alamitos,

CA, 31–40. DOI:https://doi.org/10.1145/2556288.2557241

[35] Philipp Koytek, Charles Perin, Jo Vermeulen, Elisabeth André, and Sheelagh Carpendale. 2017. MyBrush: Brushing

and linking with personal agency. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 605–615.

[36] Bill Kules and Robert Capra. 2012. Inuence of training and stage of search on gaze behavior in a library catalog

faceted search interface. Journal of the American Society for Information Science and Technology 63, 1 (2012), 114–138.

DOI:https://doi.org/10.1002/asi.21647

[37] Jelmer Kuurstra. 2015. Individual Dierences in Human-Computer Interaction: A Review of Empirical Studies. University

of Twente.

[38] Alexandra Kuznetsova, Per B. Brockho, and Rune Haubo Bojesen Christensen. 2017. lmerTest package: Tests in

linear mixed eects models. Journal of Statistical Software 82, 13 (2017), 1–26.

[39] Sébastien Lallé, Cristina Conati, and Roger Azevedo. 2018. Prediction of student achievement goals and emotion

valence during interaction with pedagogical agents. In Proceedings of the 17th International Conference on Autonomous

Agents and Multiagent Systems. 1222–1231.

[40] Sébastien Lallé, Cristina Conati, and Giuseppe Carenini. 2017. Impact of individual dierences on user experience

with a visualization interface for public engagement. In Proceedings of the 2nd International Workshop on Human

Aspects in Adaptive and Personalized Interactive Environments. ACM, New York, NY, 247–252. DOI:https://doi.org/10.

1145/3099023.3099055

[41] Sebastien Lallé, Cristina Conati, and Dereck Toker. 2019. A gaze-based experimenter platform for designing and

evaluating adaptive interventions in information visualizations. In Proceedings of the 11th ACM Symposium on Eye

Tracking Research and Applications. ACM, New York, NY, 60. DOI:https://doi.org/10.1145/3314111.3322502

[42] Sébastien Lallé, Dereck Toker, and Cristina Conati. 2019. Gaze-driven adaptive interventions for magazine-style nar-

rative visualizations. IEEE Transactions on Visualization and Computer Graphics 27, 6 (2019), 2491–2952. DOI:https:

//doi.org/10.1109/TVCG.2019.2958540

[43] Shahid Latif, Kaidie Su, and Fabian Beck. Authoring Combined Textual and Visual Descriptions of Graph Data. Euro-

graphics Association.

[44] Yu Li, Carla Allen, and Chi-Ren Shyu. 2019. Quantifying and understanding the dierences in visual activities with

contrast subsequences. In Proceedings of the 11th ACM Symposium on Eye Tracking Research and Applications.ACM,

New York, NY, Article 42.

[45] Adam V. Maltese, Joseph A. Harsh, and Dubravka Svetina. 2015. Data visualization literacy: Investigating data inter-

pretation along the novice—Expert continuum. Journal of College Science Teaching 45, 1 (2015), 84–90.

[46] Oludamilare Matthews, Sukru Eraslan, Victoria Yaneva, Alan Davies, Yeliz Yesilada, Markel Vigo, and Simon Harper.

2019. Combining trending scan paths with arousal to model visual behaviour on the web: A case study of neurotyp-

ical people vs people with autism. In Proceedings of the 27th ACM Conference on User Modeling, Adaptation, and

Personalization. 86–94.

[47] Alejandra Meneses, José-Pablo Escobar, and Soledad Véliz. 2018. The eects of multimodal texts on science reading

comprehension in Chilean fth-graders: Text scaolding and comprehension skills. International Journal of Science

Education 40, 18 (2018), 2226–2244. DOI:https://doi.org/10.1080/09500693.2018.1527472

[48] Ronald Metoyer, Qiyu Zhi, Bart Janczuk, and Walter Scheirer. 2018. Coupling story to visualization: Using textual

analysis as a bridge between data and interpretation. In Proceedings of the 23rd International Conference on Intelligent

User Interfaces. ACM, New York, NY, 503–507. DOI:https://doi.org/10.1145/3172944.3173007

[49] Tamara Munzner. 2014. Visualization Analysis and Design. CRC Press, Boca Raton, FL.

[50] Unaizah Obaidellah, Tanja Blascheck, Drew T. Guarnera, and Jonathan Maletic. 2020. A ne-grained assessment

on novice programmers’ gaze patterns on pseudocode problems. In ACM Symposium on Eye Tracking Research and

Applications.ACM,NewYork,NY,1–5.DOI:https://doi.org/10.1145/3379156.3391982

[51] Heather L. O’Brien, Rebecca Dickinson, and Nicole Askin. 2017. A scoping review of individual dierences in infor-

mation seeking behavior and retrieval research between 2000 and 2015.

Library & Information Science Research 39, 3

(2017), 244–254.

[52] Yasmina Okan, Mirta Galesic, and Rocio Garcia-Retamero. 2016. How people with low and high graph literacy process

health graphs: Evidence from eye-tracking. Journal of Behavioral Decision Making 29, 2-3 (2016), 271–294. DOI:https:

//doi.org/10.1002/bdm.1891

[53] Kristien Ooms, Philippe De Maeyer, and Veerle Fack. 2014. Study of the attentive behavior of novice and expert map

users using eye tracking. Cartography and Geographic Information Science 41, 1 (2014), 37–54. DOI:https://doi.org/10.

1080/15230406.2013.860255

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

Eect of Adaptive Guidance and Visualization Literacy on Gaze Aentive Behaviors 28:45

[54] Alvitta Ottley. 2016. Toward Personalized Visualizations. Tufts University.

[55] Erol Ozcelik, Ismahan Arslan-Ari, and Kursat Cagiltay. 2010. Why does signaling enhance multimedia learning?

Evidence from eye movements. Computers in Human Behavior 26, 1 (2010), 110–117. DOI:https://doi.org/10.1016/j.

chb.2009.09.001

[56] Matthew D. Plumlee and Colin Ware. 2006. Zooming versus multiple window interfaces: Cognitive costs of visual

comparisons. ACM Transactions on Computer-Human Interaction 13, 2 (2006), 179–209. DOI:https://doi.org/10.1145/

1165734.1165736

[57] Alex Poole and Linden J. Ball. 2006. Eye tracking in HCI and usability research. In Encyclopedia of Human Computer

Interaction. IGI, 211–219.

[58] George E. Raptis, Christina Katsini, Marios Belk, Christos Fidas, George Samaras, and Nikolaos Avouris. 2017. Using

eye gaze data and visual activities to infer human cognitive styles: Method and feasibility studies. In Proceedings of

the 25th Conference on User Modeling, Adaptation, and Personalization. 164–173.

[59] K. Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin

124, 3 (1998), 372–422. DOI:https://doi.org/10.1037/0033-2909.124.3.372

[60] Jonathan C. Roberts. 2007. State of the art: Coordinated and multiple views in exploratory visualization. In Proceedings

of the 5th International Conference on Coordinated and Multiple Views in Exploratory Visualization (CMV’07). IEEE, Los

Alamitos, CA, 61–71.

[61] Darrell S. Rudmann, George W. McConkie, and Xianjun Sam Zheng. 2003. Eyetracking in cognitive state detection

for HCI. In Proceedings of the 5th International Conference on Multimodal Interfaces. ACM, New York, NY, 159–163.

[62] E. Segel and J. Heer. 2010. Narrative visualization: Relling stories with data. IEEE Transactions on Visualization and

Computer Graphics 16, 6 (2010), 1139–1148. DOI:https://doi.org/10.1109/TVCG.2010.179

[63] Frank Serani. 2012. Reading multimodal texts in the 21st century. Research in the Schools 19, 1 (2012), 26–32.

[64] Nelson Silva, Tobias Schreck, Eduardo Veas, Vedran Sabol, Eva Eggeling, and Dieter W. Fellner. 2018. Leveraging

eye-gaze and time-series features to predict user interests and build a recommendation model for visual analysis. In

Proceedings of the 2018 ACM Symposium on Eye Tracking Research and Applications. ACM, New York, NY, Article 13,

9 pages.

[65] Ben Steichen, Michael M. A. Wu, Dereck Toker, Cristina Conati, and Giuseppe Carenini. 2014. Te,Te,Hi,Hi: Eye gaze

sequence analysis for informing user-adaptive information visualizations. In Proceedings of the 22nd International

Conference on User Modeling, Adaptation, and Personalization. 183–194. DOI:https://doi.org/10.1007/978-3-319-08786-

3_16

[66] Ben Steichen, Michael M. A. Wu, Dereck Toker, Cristina Conati, and Giuseppe Carenini. 2014. Te,Te,Hi,Hi: Eye gaze

sequence analysis for informing user-adaptive information visualizations. In Proceedings of the 22nd International

Conference on User Modeling, Adaptation, and Personalization. 183–194. DOI:https://doi.org/10.1007/978-3-319-08786-

3_16

[67] Markus Steinberger, Manuela Waldner, Marc Streit, Alexander Lex, and Dieter Schmalstieg. 2011. Context-preserving

visual links. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2249–2258. DOI:https://doi.org/

10.1109/TVCG.2011.183

[68] Robert H. Tai, John F. Loehr, and Frederick J. Brigham. 2006. An exploration of the use of eye-gaze tracking to study

problem-solving on standardized science assessments. International Journal of Research & Method in Education 29, 2

(2006), 185–208.

[69] Rong Tang and Yeseul Song. 2018. Cognitive styles and eye movement patterns: An empirical investigation into user

interactions with interface elements and visualisation objects of a scientic information system. Information Research

23, 2 (2018).

[70] Dereck Toker and Cristina Conati. 2014. Eye tracking to understand user dierences in visualization processing with

highlighting interventions. In Proceedings of the 22nd International Conference on User Modeling, Adaptation, and

Personalization. 219–230. DOI:https://doi.org/10.1007/978-3-319-08786-3_19

[71] Dereck Toker, Cristina Conati, and Giuseppe Carenini. 2019. Gaze analysis of user characteristics in magazine style

narrative visualizations. User Modeling and User-Adapted Interaction 29 (2019), 977–1011.

[72] Dereck Toker, Cristina Conati, Giuseppe Carenini, and Mona Haraty. 2012. Towards adaptive information visualiza-

tion: On the inuence of user characteristics. In Proceedings of the 20th International Conference on User Modeling,

Adaptation, and Personalization. 274–285.

DOI:https://doi.org/10.1007/978-3-642-31454-4_23

[73] Dereck Toker, Cristina Conati, Ben Steichen, and Giuseppe Carenini. 2013. Individual user characteristics and infor-

mation visualization: Connecting the dots through eye tracking. In Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems. ACM, New York, NY, 295–304. DOI:https://doi.org/10.1145/2470654.2470696

[74] Tamara Van Gog. 2014. The signaling (or cueing) principle in multimedia learning. In The Cambridge Handbook of

Multimedia Learning (2nd ed.). Cambridge University Press, New York, NY, 263–278. DOI:https://doi.org/10.1017/

CBO9781139547369.014

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.

28:46 O. Barral et al.

[75] Julia M. West, Anne R. Haake, Evelyn P. Rozanski, and Keith S. Karn. 2006. eyePatterns: Software for identifying

patterns and similarities across xation sequences. In Proceedings of the 2006 Symposium on Eye Tracking Research

and Applications (ETRA’06). ACM, New York, NY, 149–154. DOI:https://doi.org/10.1145/1117309.1117360

[76] Daricia Wilkinson, Moses Namara, Karla Badillo-Urquiola, Pamela J. Wisniewski, Bart P. Knijnenburg, Xinru Page,

Eran Toch, and Jen Romano-Bergstrom. 2018. Moving beyond a “one-size ts all” exploring individual dierences in

privacy. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems.1–8.

[77] Qiyu Zhi, Alvitta Ottley, and Ronald Metoyer. 2019. Linking and layout: Exploring the integration of text and visual-

ization in storytelling. Computer Graphics Forum 38, 3 (2019), 675–685. DOI:https://doi.org/10.1111/cgf.13719

[78] Mary C. Dyson and Mark Haselgrove. 2001. The inuence of reading speed and line length on the eectiveness of

reading from screen. International Journal of Human-Computer Studies 54, 4 (2001), 585–612. https://doi.org/10.1006/

ijhc.2001.0458

Received July 2020; revised December 2020; accepted January 2021

ACM Transactions on Interactive Intelligent Systems, Vol. 11, No. 3-4, Article 28. Publication date: August 2021.